Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support to append to existing IPC Arrow file #972

Merged
merged 1 commit into from
Apr 30, 2022
Merged

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Apr 30, 2022

This PR adds support to append new record batches to an existing Arrow IPC file.

The implementation is simple:

  1. read the file's metadata
  2. seek to the end of the last block
  3. write to it

Closes #963 - thanks to @illumination-k for fielding the idea!

@jorgecarleitao jorgecarleitao added the feature A new feature label Apr 30, 2022
@codecov
Copy link

codecov bot commented Apr 30, 2022

Codecov Report

Merging #972 (c4da037) into main (3810da5) will decrease coverage by 0.01%.
The diff coverage is 69.23%.

@@            Coverage Diff             @@
##             main     #972      +/-   ##
==========================================
- Coverage   71.51%   71.50%   -0.02%     
==========================================
  Files         355      356       +1     
  Lines       19631    19671      +40     
==========================================
+ Hits        14040    14065      +25     
- Misses       5591     5606      +15     
Impacted Files Coverage Δ
src/io/flight/mod.rs 0.00% <0.00%> (ø)
src/io/ipc/mod.rs 0.00% <ø> (ø)
src/io/ipc/write/mod.rs 92.30% <ø> (ø)
src/io/ipc/append/mod.rs 60.00% <60.00%> (ø)
src/io/ipc/write/common.rs 82.45% <90.90%> (+0.80%) ⬆️
src/io/ipc/read/reader.rs 75.79% <100.00%> (ø)
src/io/ipc/write/file_async.rs 75.29% <100.00%> (ø)
src/io/ipc/write/stream.rs 73.33% <100.00%> (ø)
src/io/ipc/write/stream_async.rs 72.41% <100.00%> (ø)
src/io/ipc/write/writer.rs 86.36% <100.00%> (ø)
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3810da5...c4da037. Read the comment docs.

@ritchie46
Copy link
Collaborator

Wow, cool!

@jorgecarleitao jorgecarleitao merged commit fa34796 into main Apr 30, 2022
@jorgecarleitao jorgecarleitao deleted the append branch April 30, 2022 15:10
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider adding append options in IPC writer
2 participants